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Abstract 

We consider the celebrated Blackwell Approachability Theorem for two-player games with vector 
payoffs. We show that Blackwell's result is equivalent, via efficient reductions, to the existence of "no- 
regret" algorithms for Online Linear Optimization. Indeed, we show that any algorithm for one such 
problem can be efficiently converted into an algorithm for the other. We provide a useful application of 
this reduction: the first efficient algorithm for calibrated forecasting. 
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1 Introduction 



A typical assumption in game theory, and indeed in most of economics, is that an agent's goal is to optimize 
a scalar-valued payoff function-a person's wealth, for example. Such scalar-valued utility functions are 
the basis for much work in learning and Statistics too, where one hopes to maximize prediction accuracy 
or minimize expected loss. Towards this end, a natural goal is to prove a guarantee on some algorithm's 
minimum expected payoff (or maximum reward). 

In 1956, David Blackwell posed an intriguing question: what guarantee can we hope to achieve when 
playing a two-player game with a vector-valued payoff, particularly when the opponent is potentially an 
adversary? For the case of scalar payoffs, as in a two-player zero-sum game, we already have a concise 
guarantee by way of Von Neumann's minimax theorem: either player has a fixed oblivious strategy that is 
effectively the "best possible", in that this player could do no better even with knowledge of the opponent's 
randomized strategy in advance. This result is equivalent to strong duality for linear programming. 

When our payoffs are non-scalar quantities, it does not make sense to ask "can we earn at least xT\ 
Instead, we would like to ask "can we guarantee that our vector payoff lies in some convex set 5"? In 
this case, the story is more difficult, and Blackwell observed that an oblivious strategy does not suffice — in 
short, we do not achieve "duality" for vector-payoff games. What Blackwell was able to prove is that this 
negative result applies only for one-shot games. In his celebrated Approachability Theorem [3], one can 
achieve a duality statment in the limit when the game is played repeatedly, where the player may learn from 
his opponent's prior actions. Blackwell actually constructed an algorithm (that is, an adaptive strategy) with 
the guarantee that the average payoff vector "approaches" S, hence the name of the theorem. 

Blackwell Approachability has the flavor of learning in repeated games, a topic which has received 
much interest. In particular, there are a wealth of recent results on so-called no-regret learning algorithms 
for making repeated decisions given an arbitrary (and potentially adversarial) sequence of cost functions. 
The first no-regret algorithm for a "discrete action" setting was given in a seminal paper by James Hannan 
in 1956 ifTOl . That same year, David Blackwell pointed out that his Approachability result leads, as a 
special case, to an algorithm with essentially the same low-regret guarantee proven by Hannan. 

Blackwell thus found an intriguing connection between repeated vector-payoff games and low-regret 
learning, a connection that we shall explore in greater detail in the present work. Indeed, we will show 
that the relationship goes much deeper than Blackwell had originally supposed. We prove that, in fact, 
Blackwell's Approachability Theorem is equivalent, in a very strong sense, to no-regret learning, for the 
particular setting of so-called "Online Linear Optimization". Precisely, we show that any no-regret algorithm 
can be converted into an algorithm for Approachability and vice versa. This is algorithmic equivalence is 
achieved via the use of conic duality: if our goal is low-regret learning in a cone K, we can convert this into 
a problem of approachability of the dual cone K°, and vice versa. 

This equivalence provides a range of benefits and one such is "calibrated forecasting". The goal of a 
calibrated forecaster is to ensure that sequential probability predictions of repeated events are "unbiased" in 
the following sense: when the weatherman says "30% chance of rain", it should actually rain roughly three 
times out of ten. The problem of calibrated forecasting was reduced to Blackwell's Approachability Theo- 
rem by Foster [7 ], and a handful of other calibration techniques have been proposed, yet none have provided 
any efficiency guarantees on the strategy. Using a similar reduction from calibration to approachability, and 
by carefully constructing the reduction from approachability to online linear optimization, we achieve the 
first efficient calibration algorithm. 

Related work There is by now vast literature on all three main topics of this paper: approachability, online 
learning and calibration, see @ for an excellent exposition. The relation between the three areas is not as 
well-understood. 
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Blackwell himself noted that approachability implies no regret algorithms in the discrete setting. How- 
ever, as we show hereby, the full power of approachability extends to a much more general framework of 
online linear optimization, which has only recently been explored (see lfl2l for a survey) and shown to give 
the first efficient algorithms for a host of problems (e.g. HHH). Perhaps more significant, we also prove the 
reverse direction - online linear optimization exactly captures the power of approachability. Previously, it 
was considered by many to be strictly stronger than regret minimization. 

Calibration is a fundamental notion in prediction theory and has found numerous applications in eco- 
nomics and learning. Dawid Q was the first to define calibration, with numerous algorithms later given by 
Foster and Vohra [8], Fudenberg and Levine [9], Hart and Mas-Colell [11] and more. Foster has given a 
calibration algorithm based on approachability Q. There are numerous definitions of calibration in the lit- 
erature, mostly asymptotic. In this paper we give precise finite-time rates of calibration and show them to be 
optimal. Furthermore, we give the first efficient algorithm for calibration: attaining e-calibration (formally 
defined later) required a running time of polyQ-) for all previous algorithms, whereas our algorithm runs in 
time proportional to log ±. 

2 Preliminaries 

2.1 Blackwell Approachability 

A vector-valued game is defined by a pair of convex compact sets ^cfjc M m and a biaffine mapping 
£ : X x y M. d ; that is, for any a G [0, 1] and any xi,X2 G X, yi,y2 G y, we have £(ax± + (1 — 
a)x 2 , y) = a£(xi, y) + (1 - a)£(x 2 , y), and £{x, ayi + (1 - a)y 2 ) = a£(x, yi) + (1 - a)£(x, y 2 ). We 
consider £(x, y) to be the "payoff vector" when Player 1 plays strategy x and Player 2 plays strategy y. We 
consider this game from the perspective of Player 1, whom we will often refer to as "the player", while we 
refer to Player 2 as "the adversary". 

In a scalar-valued game, the natural question to ask is "how much can a player expect to gain/lose 
(in expectation) against a worst-case adversary?" With d-dimensional payoffs, of course, we don't have 
a notion of 'more' or 'less', and hence this question does not make sense. As Blackwell pointed out Q, 
the natural question to consider is "can we guarantee that the payoff vector lies in a given (convex) set S?" 
Notice that this formulation dovetails nicely with the original goal in scalar-valued game, in the following 
way. Take any halfspace H C M. , where H is parameterized by a vector v and a constant c, namely 
H = {z G M rf : z • v > c}. Then the question "can we guarantee that £(•, •) lies in H?" is equivalent to "can 
the player expect to gain at least c in the scalar- valued game defined by ^"(x, y) := £{x, y) • v?" 

Given that we would like to receive payoff vectors that lie within S, let us define three separate notions 
of achievement towards this goal. 

Definition 1. Let S be a convex set ofM. d . 

• We say that a set S is satisfiable if there exists a strategy x G X such that for any y G y, £(x, y) G S. 

• We say that a set is S half space-satisfiable if for any halfspace H D S, H is satisfiable. 

• We say that a set is S is response-satisfiable if for any y G y, there exists a x y G X such that 
£(x y ,y)G5. 

Among these three conditions the first, satisfiability, is the strongest. Indeed, it says that the player 
has an oblivious strategy which always provides the desired guarantee, namely that the payoff is in S. 
The second condition, response-satisfiability, is much weaker and says we can achieve the same guarantee 
provided we observe the opponent's strategy in advance. We will also make use of the final condition, 
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halfspace-satisfiability, which is also a weak condition, although we shall show it is equivalent to response- 
satisifiability. 

Of course, a scalar-valued game is a particular case of a vector-valued game. What is interesting is that, 
for this special case, the condition of satisfiability is in fact no stronger than response-satisfiability for the 
case when S has the form [c, oo). Indeed, this fact can be view as the celebrated Minimax Theorem. 

Theorem 1 (Von Neumann's Minimax Theorem [14J). For X and y the n-dimensional and m-dimensional 
probability simplexes, and with scalar-valued £(■, ■), the set S = [c, oo) is satisfiable if and only if it is 
response- satisfiable. 

We will also make use of a more general version of the Minimax Theorem, due to Maurice Sion. 

Theorem 2 (Sion, 1958 021). Given convex compact sets IcK n JC M. m , and a function f : Xxy — ^ M 
convex and concave in its first and second arguments respectively, we have 

inf sup /(x, y) = sup inf /(x, y) 

One might hope that the analog of Theorem [T] for vector- valued games would also hold true. Unfortu- 
nately, this is not the case. Consider the following easy example: X = y := [0, 1], the payoff is simply 
£(x,y) := (x,y) for x,y 6 [0,1], and the set in question is S := {(z,z) Vz € [0,1]}. Response- 
satisfiability is easy to establish, simply use the response strategy x y = y. But satisfiability can not be 
achieved: there is certainly no generic x for which (x, y) G S for all y. 

At first glance, it seems unfortunate that we can not achieve a similar notion of duality for games with 
vector- valued payoffs. What Blackwell showed, however, is that the story is not quite so bad: we can obtain a 
version of Theorem[T]for a weaker notion of satisfiability. In particular, Blackwell proved that, so long as we 
can play this game repeatedly, then there exists an adaptive algorithm for playing this game that guarantees 
satisfiability for the average payoff vector in the limit. Blackwell coined the term approachability. 

Definition 2. Consider a vector-valued game £(■, •) and a convex set S. Imagine we have some "learning" 
algorithm A which, given a sequence yi,y2, • • • £ y, produces a sequence xi,X2, ... via the rule xj <— 
-4(yii y2, • • • , yt-i)- For any T defineQthe distance of A to be 



\ t=i / 

where here di st (, ) mean the usual notion ^-distance between a point and a set. For a given vector-valued 
game and a convex set S, we say that S is approachable if there exists a learning algorithm A such that, 



Approachability is a curious property: it allows the player to repeat the game and learn from his op- 
ponent, and only requires that the average payoff satisfy the desired guarantee in the long run. Blackwell 
showed that response-satisfiability, which does not imply satisfiability, does imply approachability. 

Theorem 3 (Blackwell's Approachability Theorem [31). Any closed convex set S is approachable if and 
only if it is response-satisfiable. 

This version of the theorem, which appears in Evan-Dar et al. O, is not the one usually attributed 
to Blackwell, although this is essentially one of his corollaries. His main theorem states that halfspace- 
satisfiability, rather than response-satisfiability, implies satisfiability. However, these two weaker satisfiabil- 
ity conditions are equivalent: 

'We may simply write Dt{A) when S and yi , . . . , yr are clear from context. 




limsup-DT^^yi, • • • ,Yt) = 



for any sequence yi,y2, • • ■ £ y 



T-5-oo 
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Lemma 1. Given a biaffine function £(■, •) and any closed convex set S, S is response-satisfiable if and only 
if it is halfspace-satisfiable. 

Proof. We will show each direction separately. 

=^ Assume that S is response-satisfiable. Hence, for any y there is an x y such that £(x y , y) G S. Now 
take any halfspace H D S parameterized by 6, c, that is H = {z : (0, z) < c}. Then let us define a 
scalar-valued game with payoff function 

/(x,y) = <0,£(x,y)). 

Notice that H D S implies that 6 • z < c for all z G 5. Combining this with the definition of 
response-satisfiability, we see that 

sup inf /(x,y) < sup/(x y ,y) < c. 

By Sion's Theorem (Theorem |2j), it follows that inf xe ^ sup yg 3; /(x, y) < c. By compactness of 
X, we can choose the minimizer x* of this optimization. Notice that, for any y G y, we have that 
/(x*, y) < c by construction, and hence £(x*,y) £ H. H is thus satisfiable, as desired. 

-<= Assume that 5 is not response-satisfiable. Hence, there must exists some yo such that £(x, yo) ^ 5 
for every x. Consider the set U := {£(x, yo) for all x G X} and notice that J7 is convex since X is 
convex and £(■, yo) is affine. Furthermore, because S is convex and S n C/ = by assumption, there 
must exist some halfspace dividing the two, that is S C H and H C\U = 0. By construction, we 
see that for any x, £(x, yo) ^ H and hence if is not satisfiable. It follows immediately that S is not 
halfspace-satisfiable. 

□ 

For the sake of simplicity, and for natural connection to the Minimax Theorem, we prefer Theorem [3] 
However, for certain results, it will be preferable to appeal to the halfspace-satisfiablility condition instead. 



2.2 Online Linear Optimization 

In the setting Online Linear Optimization, the "learner" makes decisions from a bounded convex decision 
set K, in some Hilbert space. On each of a sequence of rounds, the decision maker chooses a point Xf G /C, 
and is then given a linear cost function f t G F, where F is some bounded set of cost functions, and cost 
(ft, x t ) is paid. The standard measure of performance in this setting, called regret, is defined as follows. 

Definition 3. The regret of learning algorithm L is 

T T 



Regret T (L) = max 

fl,...,fT£j-" 



y\ft,x t ) - minyVftt,: 
* — ' xeK * — ' 

t=i t=i 



We say that L is a no-regret learning algorithm when it holds that Regret^(i) = 0(yT) = o(T). 

We state a well-known result: 

Theorem 4. For any bounded decision set K, C H, there exists a no-regret algorithm on tC. 

Later in this paper, we shall use the Gradient Descent algorithm of Zinkevich lfl6l . Ultimately, our goal 
will be to show that this theorem is equivalent to Theorem [3] 
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2.3 Convex cones in Hilbert space 

Definition 4. A set X C M. d is a cone if it is closed under addition and multiplication by nonnegative 
scalars. Given any set K C H, define cone(K) : = {ax : a G M + ,x G K}, which is a cone in %. Also, 
given any set in Hilbert space C C T~L, we can define the polar cone ofC as 

C° := {6 G H : (0,x> < Ofor all x G C} 

We state a few simple facts on convex sets: 

Lemma 2. If C is a convex cone then (I) (C°)° = C and (2) supporting hyperplanes in C° correspond to 
points x G C, and vice versa. That is, given any supporting hyperplane H of C , H can be written exactly 
as {0 G M. d : (6, x) = 0}for some vector x G C that is unique up to scaling. 

The distance to a cone can conveniently be measure via a "dual formulation," as we now show. 

Lemma 3. For every convex cone C in Hilbert space 

distfx, C)= max (0,x) (1) 

06C°,||6»||<1 

We need to measure distance to AC after we make it into a cone via "lifting". 

Lemma 4. Consider a convex set AC C % in Hilbert space and x AC. Let ||AC|| := max y6 )c ||y ||. Define C 
to be the cone generated by the lifting offC, that is C = cone({l} © /C). Then 

dist(l © x, C) < dist(x, AC) < (1 + \\JC\\) ■ dist(l © x, C) (2) 



3 Duality of Approachability and Low-Regret Learning 



We recall the notion of an approachability algorithm A from Section 2. 1 Formally, we imagine A as a func- 
tion that observes a sequence of opponent plays yi, . . . , y t _i G y and chooses x. t <— «4(yi, 3% • • • , yt-i) 
from X, with the goal that ^ 2~2t=i ^( x t> Yt) approaches a convex set S. The convex decision sets X , y, the 
payoff function £(■, •), and the set S are all known in advance to A, and hence we may also write A^s for 
the algorithm tuned for these particular choices. 

Equivalently, we consider a no-regret algorithm L as a function that observes a sequence of linear cost 
functions fi, . . . , ft-i and returns a point xt <— L(fi, . . . , f*-i) from the decision set AC. The goal here is to 
achieve the regret Ylt=i (ft) x t) — mm xeA: 2~2t=i (ft) x ) triat i s sublinear in T. The bounded convex set AC is 
known to the algorithm in advance, and hence we may write A/c for the algorithm tuned for this particular 
set AC. 

We now prove two claims, showing the equivalence of Blackwell and Online Linear Optimization. 
Precisely what we will show is the following. Assume we are given (A) an instance of an Online Linear 
Optimization problem and (B) an algorithm that achieves the goals of Blackwell's Approachability Theorem. 
Then we shall show that we can convert the algorithm for (B) to achieve a no-regret algorithm for (A). 

Lemma 5. For S defined in Algorithm [7] there exists an approachability algorithm A for S; that is, 
D T (A; S)^0asT^oo. 

Proposition 1. The reduction defined in Algorithm^ for any input algorithm A, produces an OLO algo- 
rithm L such that Reg ^ t(£) < (1 + \\K,\\)D T (A). 
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Algorithm 1 Reduction of Approachability Alg. A to Online Linear Optimization Alg. L 
Input: Convex decision set K, C W 1 
Input: Sequence of cost functions fi, f2, . . . , fr € M d 
Input: Approachability algorithm A 

Set: Two-player vector-payoff game £:lCxR d ^ R d+l as £(x, f) = (f, x) © -f 
Set: Approach set S := cone(l © /C)° 
fori = l,...,Tdo 

Let: L(f 1 ,...,f t _ 1 ) := ^,s(fi, . . . ,f t _i) 

Receive: cost function f 4 
end for 



Algorithm 2 Conversion of Online Linear Optimization Alg. L to Approachability Alg. A 
Input: Convex compact decision sets X C W 1 and ^ C M m 
Input: Biaffine vector-payoff function £(■,■): X x y ^ M. d 
Input: Approaching Set 5c{l}x R** -1 
Input: Online Linear Optimization algorithm L 
Set: /C = cone(5)°nBi 
fori = l,...,Tdo 

Query: 6 t <- L/c(fi, f t -i), where f s < ^(x s ,y s ) 

Compute: x 4 G so that £(x 4 , y)) < for any y G J // Half space oracle 
Let: X(yi, . . . ,yt-i) := *t 
Receive: y t G y 
end for 



Now onto the second reduction. The construction in Algorithm[2]attempts the following. Assume we are 
given (A) an instance of a vector-payoff game and an approach set S and (B) a low -regret OLO algorithm. 
Then we shall show that we can convert the algorithm for (B) to achieve approachability for (A). 

Proposition 2. The reduction in Algorithm [2] produces an approachability algorithm A with distance 
bounded by 

^)<(l + l|S||)«l 
as long as S is halfspace-satisfiable with respect to £(■, •). 

4 Efficient Calibration via Approachability and OLO 

Imagine a sequence of binary outcomes, say 'rain' or 'shine' on a given day, and imagine a forecaster, say 
the weatherman, that wants to predict the probability of this outcome on each day. A natural question to ask 
is, on the days when the weatherman actually predicts "30% chance of rain", does it actually rain (roughly) 
30% of the time? This exactly the problem of calibrated forecasting which we now discuss. 

There have been a range of definitions of calibration given throughout the literature, some equivalent 
and some not, but from a computational viewpoint there are significant differences. We thus give a clean 
definition of calibration, first introduced by Foster [7 ], which is convenient to asses computationally. 

We let j/2) •• ■ £ {0, 1} be a sequence of outcomes, and pi,P2> • • • € [0, 1] a sequence of probability 
predictions by a forecaster. We define for every T and every probability interval [a, b], where < a < b < 1, 
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the quantities 



n T (p,e) := > % € (p - e/2, p + e/2) , Pt(p,£J := — ^ ; ; • 

nT{P,£) 

The quantity px{p — e/2,p + e/2) should be interpreted as the empirical frequency of yt = 1, up to round 
T, on only those rounds where the forecaster's prediction was "roughly" equal to p. The goal of calibration, 
of course, is to have this empirical frequency pr(p, e) be close to the estimated frequency p, which leads us 
to the following definition. 

Definition 5. Let the (t\, e)-calibration rate for forecaster A be 

CfM)= X, r be-p T (ae,e)|-- 

8=0 

Wfe say ?/iaf a forecaster is (£i, e)-calibrated ifC^(A) = o(l). 77zw in implies lim sup^^^ (^4) = 0. 

This definition emphasizes that we can ignore an interval (p — e/2, p + e/2) in cases when our forecaster 
"rarely" makes predictions within this interval — more precisely, when we forecast within this interval with a 
frequency that is sublinear in T. Another important feature of this definition is the constant e/2 - which is an 
artifact of the discretization by e. This is the smallest constant which allows for lim sup^^ Cj,(A) = 0. 

We given an equivalent and alternative characterization of this definition: let the calibration vector at 
time T denoted ct be given by: ct{i) = "^^f'^ \ie — pr(ie, e)| 

Claim 1. The (£i, e) -calibration rate is equal to the distance of the calibration vector to the l\ball of radius 
e/2: 

C e T = dist(c T ,Bi(e/2)) 



Proof. Notice that: 



disti(x, B\(e/2)) := min ||x — y||i = — e/2 + ||x||i 

y:||y||i<e/2 



where the second equality follows by noting that an optimally chosen y will lie in the same quadrant as 
x. □ 

A standard reduction in the literature (see e.g. El) shows that e-calibration and full calibration are 
essentially the same (in the sense that an e-calibrated algorithm can be converted to a calibrated one). For 
simplicity we consider only e-calibration henceforth. 



4.1 Existence of Calibrated Forecaster via Blackwell Approachability 

A surprising fact is that it is possible to achieve calibration even when the outcome sequence {yt} is chosen 
by an adversary, although this requires a randomized strategy of the forecaster. Algorithms for calibrated 
forecasting under adversarial conditions have been given in Foster and Vohra [8], Fudenberg and Levine [9], 
and Hart and Mas-Colell ifTTl . 

Interestingly, the calibration problem was reduced to Blackwell's Approachability Theorem in a short 
paper by Foster in 1999 [7]. Foster's reduction uses Blackwell's original theorem, proving that a given set 
is halfspace-satisfiable, in particular by providing a construction for each such halfspace. Here, we provide 
a reduction to Blackwell Approachability using the response-satisfiability condition, i.e. via Theorem [3] 
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which is both significantly easier and more intuitive than BlackwelQ We also show, using the reduction to 
Online Linear Optimization from the previous section, how to achieve the most efficient known algorithm 
for calibration by taking advantage of the Online Gradient Descent algorithm of Zinkevich [16], using the 
results of Section [3] 

We now describe the construction that allows us to reduce calibration to approachability. For any e > 
we will show how to construct an e) -calibrated forecaster. Notice that from here, it is straightforward 
to produce a well-calibrated forecaster [§]. For simplicity, assume e = 1/m for some positive integer m. 
On each round t, a forecaster will now randomly predict a probability p t G {0/m, 1/m, 2/m, . . . , (m — 
l)/m, 1}, according to the distribution w t , that is Pr(pt = i/m) = wt{i). We now define a vector-valued 
game. Let the player choose W( G X := A m+ i, and the adversary choose y t G y := [0, 1], and the payoff 
vector will be 

t(wt,yt) := /w*(0) (vt-^j ,w t (l) (y t -^A ,...,w t (m)(y t -l)\ (3) 

Lemma 6. Consider the vector-valued game described above and let S, the l\ ball of radius e/2. If we 
have a strategy for choosing wt that guarantees approachability of S, that is A Ylt=i ^( w t> yt) — > S, then 
a randomized forecaster that selects pt according to wt is [l\,e)-calibrated with high probability. 

The proof of this lemma is straightforward, and is similar to the construction in Foster - The vector 
^ Ylt=i ^( w t> Vt) is simply the expectation of (£i, e) -calibration vector at T. Since each p t is drawn inde- 
pendently, by standard concentration arguments we can see that if h Y^t=\ ^( w t; yt) is close to the ^iball of 
radius e/2, then the (£\, e) -calibration vector is close to the e/2 ball with high probability. 

We can now apply Theorem [3] to prove the existence of a calibrated forecaster. 

Theorem 5. For the vector-valued game defined in Q, the l\ ball of radius e/2 is response-satisfiable and, 
hence, approachable. 

Proof. To show response-satisfiability, we need only show that, for every strategy y G [0, 1] played by the 
adversary, there is a strategy w G A m for which £(w, y) G S. This can be achieved by simply setting i 
so as to minimize \ie — y\, which can always be made smaller than e/2. We then choose our distribution 
w G A m+ i to be a point mass on i, that is we set w(i) = 1 and w(j) = for all j / i. Then £{w, y) is 
identically everywhere except the ith coordinate, which has the value y — i/m. By construction, y — i/m G 
[—1/m, 1/m], and we are done. □ 



4.2 Efficient Algorithm for Calibration via Online Linear Optimization 

We now show how the results in the previous Section lead to the first efficient algorithm for calibrated 
forecasting. The previous theorem provides a natural existence proof for Calibration, but it does not imme- 
diately provide us with a simple and efficient algorithm. We proceed according to the reduction outlined in 
the previous section to prove: 

Theorem 6. There exists a {t\,e)-calibration algorithm that runs in time 0(log -) per iteration and satisfies 

The reduction developed in Proposition [2] has some flexibility, and we shall modify it for the purposes 
of this problem. The objects we shall need, as well as the required conditions, are as follows: 

1 . A convex set K, 

2 A similar existence proof was discovered concurrently by Mannor and Stoltz 1131 
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2. An efficient learning algorithm A which, for any sequence fi, f2, . . ., can select a sequence of points 
9i,02, ■ ■ ■ G 1C with the guarantee that J2t=i(^t, &t) ~ m ^ n e&K J2t=i(^t, 0) = o(T). For the reduc- 
tion, we shall set f t < £(wt, yt). 

3. An efficient oracle that can select a particular wt G X for each 0t G /C with the guarantee that 



dist 



(^E £ ( W ^)'S) < i ^(-£(w t ,2/O,^)-min^(-£(w 4 ,y t ),0)^ (4) 



where the function dist() can be with respect to any norm. 

The Setup Let K, = £?oo(l) = {0 G M. d : ||0||oo < 1} be the unit cube. This is an appropriate choice 
because we can write the ^distance to B\{e/2) (the ^iball of radius e/2) as 

disti(x, jBi(e/2)) := min llx — ylh = — e/2 + llxlli = — e/2 — min (— x, 0), (5) 

y:||y|[l<£/2 f:||e||oo<l 

where the second equality follows by noting that an optimally chosen y will lie in the same quadrant as x. 
Furthermore, we shall construct our oracle mapping h > w with the following guarantee: (£{w, y),0) < 
e/2 for any y. Using this guarantee, and if we plug in x = ^ Ylt=i £( w t, yt) @' we arr i ye at: 

dlStl ,Bi(e/2) - -e/2- e ^n( ,0 



/ T T 

Vt=i t=i 



This is precisely the necessary guarantee Q. 



Constructing the Oracle We now turn our attention to designing the required oracle in an efficient manner. 
In particular, given any with ||0||oo < 1 we must construct w G A m+ i so that (^(w, y),0) < e/2 for any 
y. The details of this oracle are given in Algorithm [3] It is straightforward why, in the final else condition, 

Algorithm 3 Constructing w from 
Input: such that ||0||oo < 1 
if 0(0) < Othen 

w <— 5q II That is, choose w to place all weight on the Oth coordinate 
else if 6(m) > then 

w <r- 5 m II That is, choose w to place all weight on the last coordinate 
else 

Binary search to find coordinate i such that 0(i) > and 0(i + 1) < 

oji)- 1 <■ , -oji+i)- 1 c 

w ^~ e(i)~ 1 -e(i+i)- 1 ° i + 0(i)-i-o{i+i)-i Oi + l 
end if 
Return w 



there must be such a pair of coordinates i, i + 1 satisfying the condition. We need not be concerned with the 
case that 0(i + 1) = 0, where we can simply define — = and — = 1 leading to w <— 5i + \. It is also clear 
that, with the binary search, this algorithm requires at most O(logm) = 0(log 1/e) computation. 
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In order to prove that this construction is valid we need to check the condition that, for any y E {0, 1}, 
(£(w, y),9) < e/2; or more precisely, YliLi 0(i)w(i) (y — -) < e/2. Recalling that m = 1/e, this is 
trivially checked for the case when 0(1) < or 0(m) > 0. Otherwise, we have 

(£(w,y),9) = 0(Q - r = — (y-±-) + Q(i+ 1) - -=ff! ±il^ — (y - Hi) 

1 1 < max(|0(i)|,|O(i + l)|) ^ < e 



0(i) _1 - 0(i + I) -1 m ~ 2 ~ 2 

The Learning Algorithm The final piece is to construct an efficient learning algorithm which leads to 
vanishing regret. That is, we need to construct a sequence of t s in the unit cube (denoted i?oo(l)) so that 

T T 

T(£ t) t )- min T(£ t ,0)=o(T), 
t=l oeB x (i) t=1 

where i t := £(w tl y t ). There are a range of possible no-regret algorithms available, but we use the one 
given by Zinkevich known commonly as Online Gradient Descent |[l6l . The details are given in Algo- 
rithm [4] This algorithm can indeed be implemented efficiently, requiring only 0(1) computation on each 

Algorithm 4 Online Gradient Descent 
Input: convex set K C M. d 
Initialize: 6\ = 
Set Parameter: 77 = 0(T -1 / 2 ) 
fori = l,...,Tdo 
Receive £ t 

0' t+ i 0t — rj£t II Gradient Descent Step 
t +i <- Project 2 (0' t+1 ,K) II L2 Projection Step 
end for 

round and 0(min{m, T}) memory. The main advantage is that the vectors £ t are generated via our oracle 
above, and these vectors are sparse, having only at most two nonzero coordinates. Hence, the Gradient 
Descent Step requires only O(l) computation. In addition, the Projection Step can also be performed in an 
efficient manner. Since we assume that t E 5^(1), the updated point 0' t+1 can violate at most two of the 
Loo constraints of the ball Soo(l). An £2 projection onto the cube requires simply rounding the violated 
coordinates into [—1,1]. The number of non-zero elements in can increase by at most two every itera- 
tion, and storing is the only state that online gradient descent needs to store, hence the algorithm can be 
implemented with 0(min{T, m}) memory. We thus arrive at an efficient no-regret algorithm for choosing 
0t- 

Proof of Theorem^ Here we have bounded the distance directly by the regret, using equation ([4]), which 
tells us that the calibration rate is bounded by the regret of the online learning algorithm. Online Gradient 
Descent guarantees the regret to be no more than DGVT, where D is the £2 diameter of the set, and G is 

the ^2-norm of the largest cost vector. For the ball B^l), the diameter D = yf^, and we can bound the 

norm of our loss vectors by G = y/2. Hence: 

= di S t(c T ,5 l(£ /2)) < ^ < 9R = o(J=) (6) 

□ 
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A Proofs 



Proof of Lemma^ We need two simple observations. Define 7rc(x) as the projection of x onto C. Then 
clearly, for any x, 

dist(x,C) = ||x-7rc(x)|| (7) 
(x - 7T C (x), y) < Vy G C and hence x - vr c (x) G C° (8) 
(x-7rc(x),7T C (x)} = (9) 

Given any 6 G C° with \\6\\ < 1, since 7rc(x) G C we have that 

(6>,x) < (0,x-tt c (x)> < ||6>||||x-7r c (x)|| < ||x-vr c (x)||, 

which immediately implies that max 0gC o ii0ii<i(0, x) < dist(x, C). Furthermore, by selecting = 
x-^cW w j 1 j c ] 1 ^ as norm one an d by dTl, is in C°, we see that 

||x-7rc(x)|| ' J 1— T' ' 

max (0,x) > ( -. —. L- ,x\ = / — | ,x — 7rc(x)\ = llx — 7Tc(x)||, 

fl6C0,||e||<i x 7 " \||x-vr c (x)||' / \||x-7r c (x)||' ° v 7 11 

which implies that max 9gC o ,|i0||<i(0, x) > dist(x, C) and hence we are done. □ 

Proof of Lemma^ Since dist(l © x, {1} x /C) = dist(x, K.) and {1} © K, C C, the first inequality 
follows immediately. 

For the second inequality, let y = 1 © x and let w, v be the closest points to y in C and K, respec- 
tively. Consider the plane determined by these three points, as depicted in figure [T] Notice that, by triangle 




Figure 1: The ratio of distances to K, and the cone is the same as the ratio between ||v|| and one. 
similarity, we have that 

mm = M = lly~ V H = dist(y, {!}©£) 
||1©0|| ||y-w|| dist(y,C) 

Of course, v G K, and hence ||v|| < ||{1} © K,\\ < 1 + ||/C||. The result follows immediately. □ 

Proof of Lemma^ The existence of an approachability algorithm is established by Blackwell's Approach- 
ability Theorem (Theorem[3]), as long as we can guaranteed the response-satisfiability condition. Precisely, 
we must show that, for any f, there is some Xf G K, such that £(xf , f) = (f , Xf ) © — f G cone(l © /C)°. 
Recall that 6 G cone(l © /C)° if and only if (9, z) < for every z G cone(l © /C). Observe that it suffices 
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to restrict to only the set generating the cone, that is 9 £ cone(l © JC)° if and only if (1 © x', z) < for 
each x' G fC. Hence, 

l(x,f)eS «f,x) © -f,z) < Vz G cone(l © 1C) 

<=> «f,x) ®-f,l©x') < Vx' G/C 
(f,x)<(f,x') Vx' G/C 

Of course, this can be achieved by setting x = arg min xg ^(f , x), and hence we are done. □ 

Proof of Proposition^ First notice that we require the halfspace-satisfiability condition for S to ensure that 
the "halfspace oracle" in Algorithm [2] exists. Because we are selecting 6 t in K, C cone(5') , 6t defines a 
halfspace containing S and hence we can use our halfspace oracle to find an X| satisying (0 t ,£(-x. t ,y)) < 
for every y G y. 

To bound Dt(A), which is the distance between the point ^ Y2t=i ^( x *i Yt) an( l tne set S> we begin by 
instead bounding the distance to cone(5). We can immediately apply Lemma[3]to obtain 

^ fe(f t ,0 t )-wmJ2&> 9 n = i Re § ret T(^) ( 10 ) 

\t=i e t=i / 



< 



where the first inequality follows by the halfspace oracle guarantee. Of course, if we let S' C M. d 1 be the 
set S after removing the first coordinate, then we see by Lemma [4] that for any z G 

dist(lffiz, S) = dist(z, S') < (l+||S'||)dist(l©z, cone(l©S")) < (l+||5||)dist(l0z, cone(5)). 

(11) 

By assumption, however, we can write A Y2t=i ^( x t) Yt) = 1 © z for some z G Combining this with 

equations ( fT0| ) and (jTTJ finishes the proof. □ 

Proof of Proposition^ Applying Lemma[3]to the definition of Dt(A) gives 

D T (A) = d±st (^^(x t ,f t ),s) = „ (^Y, e ^V'A W 

yl ^ y w6cone(le/C) , ||w||<l \i ^ / 



Notice that, in this optimization, we can assume w.l.o.g. that ||w|| = 1, or w = 0. In the former case we 



can write w = TrffrTT for some x G fC, and we drop the latter case to obtain the inequality 



, , iex \ 1 (Ef=i( f *> x t) - Ef=i( f t> x ) 



T (Et=i( f ^ x t) " Et=i( f t, x *>) > ^Regret T („4) 



,., - Jiex| 

' ^ T " x *) - El 

,U©x*|| - l + 

where we set x* := argmin xg yc Ylt=i(^t, x). 



□ 
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B A generalization of Blackwell to convex functions 

Consider the following generalization of Blackwell to functions. In analogy to Blackwell, let: 

1. A two-player game with functions as payoffs. For strategies i,j we have that the payoff of the game 

j) G S is a function (rather than a vector as in Blackwell). 

2. S - set of functions S : {/ : R d i-> R} C T. 

3. "Halfspaces", which are characterized by x G R d . The halfspace H x contains all functions / such 
that f(x) < 0. 

4. An oracle O : H x i-> p which maps a halfspace containing 5, i.e. V/ G 5, /(x) < 0, into a distri- 
bution over player strategies, such that the resulting loss function is contained inside the halfspace, 
i.e. 

Vj ,l(0(x),j) = l( P ,j)eH x 

5. When talking of approachability we need a distance measure. If we think of f(x) as the "inner- 
product" between / and x, then K = {x\f(x) < V/ G S}, which is the "dual" set to S, is the set 
of all hyperplanes containing S. Our distance measure between a function / and S is then taken to be 
the maximal inner product with any hyper-plane containing S: 

d(f, S) = max{0,max/(x)} 
Note that this distance is zero for all members of S. 

Then: 

Theorem 7 (Blackwell generalization for functions). Given an Oracle as above, the set S is approachable. 

The Blackwell method of proof is geometric in Nature, and it is not immediately clear how to generalize 
it to prove the above. However, using Online Convex Optimization, the proof is a simple generalization of 
the one in the previous sections: 

Proof. Define the dual set to S as 

K = {x\f(x) < V/ G S} 

Define the gain function for iteration t- ft - as ft = l(pt,jt) where j t is the adversary's strategy at iteration 
t, and pt is given by the oracle as the mixed user strategy for the hyperplane parameterized by xt. 

Hence, iteratively the OCO algorithm generates an x t G K, which is then fed to the Oracle to obtain 
pt = 0(xt), which in turn defines the cost function for this iteration ft = l(pt,j)- The OCO low-regret 
theorem guaranties us that 



max ■ . 

t t 



By the guarantee provided by the oracle, we have that ft{xt) < 0, which combined with the above gives 

d(f, S) = max />*) = max i £ f t (x*) < e t M- 



t 

Which by our definition implies that the distance of the average gain function to the set S converges to zero. 

□ 
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